Support more locales in R-CMD-check-occasional - #7869

• HEAD=occasional-more-locales much slower for transform improved in #5493
• HEAD=occasional-more-locales slower P<0.001 for DT[by] max regression fixed in #7480
  

Generated via commit ef77a1e

Download link for the artifact containing the test results: ↓ atime-results.zip

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.01%. Comparing base (0ef8d1f) to head (0325481).

@@           Coverage Diff           @@
##           master    #7869   +/-   ##
=======================================
  Coverage   99.01%   99.01%           
=======================================
  Files          88       88           
  Lines       17293    17293           
=======================================
  Hits        17123    17123           
  Misses        170      170           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

At least partly this is blocked by an upstream error that {R.oo} can't install in az_AZ:

HenrikBengtsson/R.oo#31

Interesting. I've tried test.data.table() in all locales again (setting LC_ALL but also LANGUAGE=en; otherwise the failures are inconvenient to parse and many of those are probably due to R being translated while data.table is not) and the only remaining failures are tests 168.[1-3] in locales bo_CN, bo_IN, dz_BT, km_KH, shn_MM, tok, vi_VN: this is due to R bug 19117. Are the tests supposed to be still failing in CI?

Surprising that the same date works in all the problem locales!

passes = function() format(as.Date(strptime(strftime(as.Date('2000-11-01'), '%Y %d %b'), '%Y %d %b'))) == '2000-11-01'

for (ll in c("bo_CN", "bo_IN", "dz_BT", "km_KH", "shn_MM", "tok", "vi_VN"))
  print(withr::with_locale(list(LC_TIME=ll), passes()))
# all FALSE

# .utf8 suffix needed for https://bugs.r-project.org/show_bug.cgi?id=19117#c1
for (ll in c("en_US", "zh_CN.utf8", "lv_LV", "es_ES", "ru_RU.utf8", "pl_PL", "th_TH.utf8"))
  print(withr::with_locale(list(LC_TIME=ll), passes()))
# all TRUE

Looking at the actual 168 series, it looks like the strptime() behavior is pretty irrelevant?

NOV = format(strptime("2000-11-01", "%Y-%m-%d"), "%b")
x = c("09:29:16","10:42:40","23:47:12","01:06:01","11:35:34","11:51:09")
datetimes = paste0("2011 ", NOV, c(18,18,18,19,19,19), " ", x)
DT = IDateTime(strptime(datetimes,"%Y %b%d %H:%M:%S"))

AFAICT {data.table} only gets involved with the POSIXlt object received from strptime() -- why wouldn't we just use 2011-11-01 ... strings directly? The comment doesn't make it clearer to me:

# IDateTime conversion methods that ggplot2 uses (it calls as.data.frame method)
# Since %b is e.g. "nov." in LC_TIME=fr_FR.UTF-8 locale, we need to
# have the target/y value in these tests depend on the locale as well, #3450.

#4719 added that, it just took the step of making NOV more LC_TIME friendly, but I don't see any discussion of why not move to ISO inputs (I'm guessing it was just done note to risk messing with back-compatibility of tests). cc @tdhock

Grr, why is lv_LV still failing?

Edit: it's the pathological case. The string vector being sorted is c(rep("yoxykqrx", 999), "mmmmmmmmmmmmmmmmm"), and they always sort differently, no matter the number of ms:

> c("yoxykqrx", "mmmmmmmm") |> order()
[1] 1 2
> c("yoxykqrx", "mmmmmmmm") |> forderv()
[1] 2 1

Edit:

x <- c("tgaaznck", "zcbkervn", "mttamosk", "rnargwdj", "ylzuozyx", "ncnzxvrz")
idx_c <- order(x, method='radix')
idx_r <- order(x, method='shell')
table(radix = x[idx_c], shell = x[idx_r])

          shell
radix      ylzuozyx mttamosk ncnzxvrz rnargwdj tgaaznck zcbkervn
  ylzuozyx        0        0        0        0        1        0
  mttamosk        1        0        0        0        0        0
  ncnzxvrz        0        1        0        0        0        0
  rnargwdj        0        0        1        0        0        0
  tgaaznck        0        0        0        1        0        0
  zcbkervn        0        0        0        0        0        1

We can remove one element to make the sort order identical. Unfortunately, the one mis-sorted element is second-to-last, so removing which(sorted_c != sorted_r) removes all of them.

For make_words(50L), the optimal subset can be found in only $2^{50}$ attempts! There has to be a heuristic to speed this up...

We had green here: https://github.com/Rdatatable/data.table/actions/runs/30947247113

Where are you seeing the error?

It's intermittent. The previous push was red for one of the locales: https://github.com/Rdatatable/data.table/actions/runs/31011336245/job/92324013081?pr=7869

Confirmed -- about 5% flakiness. Argh.

OK, make_c_sortable() wasn't quite WAI -- the intention was that, at each step, we remove one of the problematic strings. But in practice the string being removed was effectively random, hence why we could with "high" probability wind up with only 1 string left.

Once that's fixed, it becomes exceedingly unlikely to wind up with few enough strings that they are sorted after sample(). Here's 10000 iterations without it happening (tried in lv_LV and az_AZ):

make_c_sortable = function(x) {
  while (!identical(idx_c<-order(x, method='radix'),
                    idx_r<-order(x, method='shell'))) {
    sorted_c = x[idx_c]
    sorted_r = x[idx_r]

    x = x[-idx_c[which(sorted_c != sorted_r)[1L]]]
  }
  x
}

make_words <- function(n) apply(matrix(sample(letters, n*8L, TRUE), ncol=8L), 1, paste, collapse="")
for (ii in 1:10000) {
  c2 = sample(make_c_sortable(make_words(50L)), 1e3, TRUE)
  stopifnot(!identical(order(c2), seq_along(c2)))
}